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ABSTRACT 

A study was made of professional qualifications for 
personnel employed as program e valuators and auditors. These 
qualifications, according to operational or theoretical models, are 
necessary to assure local school districts of obtaining the services 
of competent and ethical personnel. Findings of (1) a review of 
literature, (2) a national survey of directors and staffs of research 
and program evaluation departments of public schools, (3) a review of 
representative contractual relations and job qualifications in use, 
(4) a survey of ten university training programs, and (5) a survey of 
legislators to determine the attributes of an evaluation report which 
make it acceptable as a basis for decision-making, are reported. 
Criteria for employment of program evaluators and auditors are 
recommended, and the political implications of an accrediting process 
are discussed. (Author/BW) 
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As the evaluation of educational programs increases in number and importance, the following 
questions have become more frequent in the literature as discussion topics among training 
institutions and as areas of concern expressed by prospective clients of evalrstors: 

1. What qualifications should personnel who label themselves educational program 
evaluators and auditors or who perform such services have? 

2. What training and experiences in educational program evaluation are being offered 
by institutions of higher learning? 

3. What evaluative criteria impact on legislators in their decision-making roles? 

The major purposes of this study were to determine the present state of thinking regarding 
qualifications for educational program evaluators and to match such findings with theoretical 
and operational criteria. Another purpose was to determine what aspects of program 
evaluation are considered important to legislators. 

The following activities were undertaken to answer the above: 

1 . A review of the literature was made. 

2. A survey of representative national public school directors and staffs of research and 
program evaluation departments was conducted to determine desirable 
qualifications. 

3. A review of selected higher education training programs for evaluators was 
conducted. 

4. Interviews with selectedprogram administrators were conducted, and a survey of the 
California Legislature and Congressional representatives was made. 

A review of recent literature revealed an array of new book titles and periodical articles related 
to educational program eva«..ation. While some writers such as Walberg (1974), offered 
practical suggestions often in the form of case studies, others were more generic and 
theoretical. The ERIC file offered a number of examples of documents which yielded exhortative 
requests for more specialized training of program evaluators. Worthen (1972) and Popham 
(1975), among others, have recently addressed the issue of evaluator qualifications directly. 
The California Society of Educational Program Auditors and Evaluators will focus on 
qualifications for educational evaluators at its conference in May 1976. 

Thesurveyed literature indicated that educational program evaluators were: (1) thrust intothe 
midst of political as well as pedagogical arenas, (2) in need of clarifying their roles, (3) in need 
of operating effectively with both lay and professional personnel, and (4) not as well equipped, 
either by training or experience, to do their jobs. In general, contemporary writers recognized 
the potential power and actual shortcomings of both program evaluators and their products. 
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{This paper was presented at the annual meeting of the American Educational Research 
Association, San Francisco, Calif., April 1976. 
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Procedure. A survey to determine desirable qualifications for personnel conducting 
educational program evaluations and audits was conducted nationally. Questionnaires were 
sent to a sampfp of 206 directors and staffs of research and program evaluation departments of 
public school LEAs and selected inte rmediate units. N inety-f ive directors and staffs of research 
departments for public school agencies in grades K-14 ir s Angeles County and 111 
directors and staffs listed on the Large City and County Test Directors' Roster distributed by 
Personnel Press, were surveyed. One hundred twenty-two (59%) usable surveys were 
returned. 

A second survey to determine the aspects of an evaluation report, which legislators deem as 
important in accepting such a report as valid and reliable and upon which they would make t 
political decision, was made of 120 California Legislators and 45 California Congressmen. 
Sixteen usable surveys were returned; another 1 6 unusable surveys were returned. Interviews 
with 24 project directors in California and 22 practicing evaluators in California were 
conducted to determine their perception of criteria for practicing educational program 
evaluators. 

Results. In the area of formal training, the respondents indicated variability as to the number 
of semester units which should be required of program evaluators. Those courses which are 
often labeled "educational foundation courses" generated two types of responses: a 
consensus for minimal training (one or two courses) toa split between minimal or no training 
needed. (See Table 1.) 

TABLE 1 

Percent of Responses for Number of College/University 
Course Semester Units Required for Program Evaluators 



Course 


Type of 
Course 


No. of Semester 
Units* 


N by 
Stem , 


3 


6 


9 


12 


None 
Needed 


1. Business 


1 F 


30 


10 


5 


4 


54 


105 


2. Child Growth and Development 


2 F 


35 


27 


16 


12 


12 


113 


3. Computer Methodology 


3T 45 


29 


12 


4 


12 


112 


4. Curriculum Development 


4 F 


24 


32 


19 


19 


8 


117 


5. Experimental Design 


5 T 


27 


38 


15 


16 


5 


116 


6. Evaluation Procedures 


6T 


14 


30 


22 


33 


2 


116 


7. History of Education 


7 F 


37 


5 


2 


0 


61 


110 


8. Instructional Methodology 


8 F 


36 


24 


20 


10 


12 


115 


9. Instructional Technology 


9 F 


38 


24 


10 


9 


20 


108 


10. Learning Theory 


10 F 


41 


27 


17 


10 


6 


115 


11. Multivariate Analysis 


11 T 


41 


29 


14 


3 


14 


109 


12. Philosop^ Education 


12 F 


56 


13 


2 


1 


32 


112 


13. Researcl - sign 


13 T 


25 


35 


19 


19 


2 


113 


14. School Administration 


14 F 


32 I 28 


9 


6 


28 


116 


1 5. School Law 


15 F 


57 


12 


1 


3 


31 


114 


16. Sociology of Education 


16 F 


47 


14 


3 


3 


38 


111 


17. Theory of Measurement 


17T 


37 


36 


15 


11 


2 


115 


18. Univariate Analysis 


18T 


52 


17 


15 


4 


13 


106 


19. Other (specify): 




24 


18 


6 


12 


7 


17 



F - Foundation course ^ ; 

T - Technical course i 




*The percentage is rounded off to the nearest whole number 
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Those foundation course items showing consensus for minimal training wr r e: Child Growth 
and Development (2), Instructional Methodology (8), lnstructionalTechno!og> {9), andLearning 
Theory (10). Foundation course items showing a split in that nearly one-third of the 
respondents favored either minimal or nocourseworkwere: Business(1 ), History of Education 
(7), Philosophy of Education (12), School Administration (14), School Law(1 5),andSociologyof 
Education (16). 

In general, there was greater consensus regarding the need for specialized or technical 
courses: Computer Methodology (3), Experimental Design (5), Evaluation Procedures (6), 
Multivariate Analysis (11), Research Design (13), School Administration (14), Theory of 
Measurement (17), and Univariate Analysis (18). Minimal to considerable training was 
indicated for all the technical courses, with the most training needed in Evaluation Procedures 
(6). Two areas of technical expertise which yielded a greater spread, particularly in the "None 
Needed" category, had to do with Computer Methodology (3) and Data Analysis (1 1 and 18). 

There was more agreement among respondents to formal degree and/or credential 
requirements than for training of program evaluators. (See Table 2.) While 91% of the 
respondents indicated that a B.A. should be mandatory, and 60% felt that an M.A. was 
mandatory, only 6% felt that a doctorate was mandatory. Forty-two percent of those surveyed 
indicated that a teaching credential was mandatory while administrative or pupil credentials 
were felt to be necessary by 14% and 10%, respectively. Thirty-two percent indicated that a 
state credential for program evaluators should be mandatory. 

The types of field experiences required of program evaluators related positively with the formal 
credential requirements. (See Table 3.) While 42% of the respondents felt that a teaching 
credential was mandatory, 44% felt that classroom teaching experience should be required. 
Similarly, where 14% would mandate an administrative credential, 19% would require 
administrative or other adult support experience. 

Experience as an interna! evaluator was deemed a requirement by 30% of the respondents 
while experience as an external evaluator was considered a requirement by 20% of those 
surveyed. External auditing experience was considered a requirement by only 12% of the 
respondents. 

Personal attributes required by educational program evaluators were ranked as 
follows: (1 ) Ability to meetproject requirements, (2) Ability tospeakandwriteappropriately for 
technical and non-technical audiences, and (3) Ability to organize tasks. (See Table4.)The top 
three ranked attributes focused on the individual compe oncies of the evaluator as opposed to 
the next set of three attributes which focused on interpersonal relationships: (4) Ability to get 
along with employer, (5) Ability to get along with professionals of equal training, and (6) Ability 
to get along with professionals of lesser technical training. 
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TABLE 2 

Percent of Responses for Degree to Which Program Evaluators 
Should Meet Formal Requirements 



Requirement 


Degree of Requirement* 


N by 
Item 


Mandatory 


Important 


Desirable Unnecessary 


20. B. A. 


91 


3 


4 


2 


1 

115 


21.M.A. 


59 


26 


12 


3 


22.Ph.D./Ed.D. 


6 


14 


57 


23 ,117 


23. Teaching Credential 


42 


19 


28 


11 


. 

- 1 Q 


24. Administrative Credential 


14 


26 


38 


22 




25. Pupil Services Credential 


10 


24 


37 


29 


119 


26. State Credential for Program 
Evaluation were one to exist 


32 


26 


27 


15 


115 


27. Other (specify): 


11 


0 


11 


78 


9 



TABLE 3 



Type of Experience 


Degree of Requirement* 


N by 


Required 


Important 


Desirable 


Unnecessary 


Item 


28. Classroom Teaching I 


44 


25 


26 


5 


120 


29. Counseling or Other Pupil 
Support Experience ! 


6 


29 


46 


19 


115 


30. Administration or Other Adult 
Support Experience ' 


19 


30 


39 


12 


117 


31. Internal Evaluator i 


30 


39 


27 


4 


119 


32. External Evaluator 


20 


41 


32 


7 


117 


33. External Auditor 


12 


32 


44 


12 


111 


34. Other (specify): 


40 


10 I 


10 


40 


10 



*The percentage is rounded off to the nearest whole number > 

i 
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Twelve percent of the respondents wrote comments in the "Other" categories. The majority of 
write-in comments on all parts of the survey primarily focused on the interpersonal skills of 
program evstuators, rather than specifying coursework or formal training experiences. 
Respondents appeared to be willing to overlook formal qualifications for proven human 
interaction skills and sensitivity to the factors operating in the process of conducting the 
evaluation of educational programs. 

A review of catalogs from 1 0 universities representing the eastern, central, and western states 
revealed that the 10 institutions offered doctorates in evaluation and research either as 
separate programs or under departments of educational psychology.{SeeTable5.)Descriptors 
in six of the universities specifically noted public schools as potential recipients of the 
specialized coursework for program evaluators for which they train. 

In a majority of cases, courses dealing with some aspects of program evaluation and/or 
research design were also included in departments other than educational psychology — most 
often, curriculum within schools of education and psychology or business/manage;ment 
outside of the schools of education. 



TABLE 4 

Ranking of Personal Attributes Which Program 
Evaluators Should Possess 



1. 


Abi 


lity to meet project requirements 


2. 


Abi 


lity to speak and write appropriately for technical ana non-technical audiences 


3. 


Abi 


lity to organize tasks 


4. 


Abi 


tity to get along with employer 


5. 


Abi 


ity to get along with professionals of lesser technical training 


6. 


Abi 


ity to get along with professionals of equal training 


7. 


Abi 


ity to write logically 


8. 


Abil 


ity to speak clearly 


9. 


Abi 


ity to write clearly 
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TABLE 5 ' 

| 

List of University Catalogs Reviewed 



Michigan State University 

Teachers College, Columbia University 
University of California at Los Angeles 

University of Chicago 

University of Colorado 

University of Connecticut 

University of Indiana 

University of Pennsylvania 

University of Southern California 

Wayne State University 



Project directors concur with the findings of the national survey for the types of formal 
degree/credential and field experiences considered desirable for educational program 
evaluators. (See Table 6.) The single most important criterion was evidence of successful 
performance prior to employment either as an internal or external evaluator , Successful was 
functionally defined as being able to perform the evaluation process in a cost-effective manner 
and being sensitive to the political and psychological factors involved. No differentiation was 
made between evaluators and auditors. 

Agreement on formal degree/ credential and field experiences with the national survey was 
also found for practicing program evaluators and auditors. (See Table 7.) Practitioners, 
however, identified formal training experiences as important for e, .jloyment or contract 
criteria more than did project directors. 

Political decision makers in the California Legislature and California Congressmen ranked the 
Reputation of the Evaluator as the most important criterion by whic*i they would make a 
political decision regarding a program based upon an evaluation report. (See Table 8.) Second 
and third rankings were the degree to which a given study corroborated previous findings and 
the degree of technicality of the report, respectively. 
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TABLE 6 

List of Agencies Contacted for Project Directors' 
Criteria for Program Evaluators 

Azusa Unified School District 

Burbank Unified School District 

Calgary Regional Office of Education 

California Commission for Teacher Preparation and Licensing 

California State Department of Education 

Downey Unified School District 

Garvey School District 

Lennox Unified School District 

Los Angeles Unified School District 

Office of the Los Angeles County Superintendent of Schools 

San Diego Community College District 

Ventura Unified School District 



TABLE 7 

List of Agencies Contacted for Practitioners' 
Criteria for Program Evaluators 

Downey Unified School District 

L. E. Orcutt and Associates 

LaVerne College 

Office of the Los Angeles County Superintendent of Schools 

University of Southern California 

Ventura Unified School District 
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TABLE 8 

Ranking of Important Aspects of an 
Educational Program Evaluation Report by 
Members of the California Legislature and 

California Congressmen (N=165) 



1. Reputation of the evaiuator 

2. Degree to which the study corroborates previous findings 

3. Degree of technicality of report 

4. Cost of the study _____ 

5. Use of comparison group(s) 

6. Utility of the study for the profession 

7. Congruence between program objectives and assessment devices 

8. Credibility of the agency the evaiuator represents 

9. Independence of the evaiuator from the project 

10. Representativeness of sample 

1 1 . Rationale of the programs to be evaluat ed 

12. Validity of the evaluation design 



Discussion and Recommendations. Asa result of the survey findings and review of the 
literature and university catalogs, the following profile of an educational program 
evaiuator emerges as being desirable: 

Training: A program evaiuator should have at least one three-unit course in the following 
foundation areas: 

Child Growth and Development 
History of Education 
Instructional Methodology 
Instructional Technology 
Learning Theory 
Philosophy of Education 
School Administration 
School Law 
Sociology of Education 

He should have at least two three-unit courses in Curriculum Development. 

He should have at least one three-unit course in the following technica! areas: 

Computer Methodology 
Multivariate Analysis 
Theory of Measurement 
Univariate Analysis 
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He should have at least two three-unit courses in Experimental Design and Research 
Design, and he should have at least three three-unit courses in Evaluation Procedures. 

Formal Requirements: A program evaluator must possess a B.A. degree and should 
possess an M.A. degree. He should be a credentia!ed teacher and hold a state credential in 
program evaluation if such a credential exists. 

Types of Field Experiences: A program evaluator should have taught in the classroom 
and had some kind of evaluation experience as either an internal or external evaluator. 

Personal Attributes: A program evaluator must be able to meet project requirements, 
have the ability to speak and write appropriately for technical and non-technical 
audiences, and be able to organize tasks related to evaluating educational programs. 

In general, there appears to be consensus by research and project directors with practicing 
evaluators as to what formal degree/credential requirements and field experiences are 
desirable. They also agree as to what personal attributes are needed. As a group, practitioners 
place more importance on technical training than do their employers. 

Politicians, while believing the worth of an educational program evaluator's report to be a 
function of the reputation of the evaluator, showed some contradiction in attributing 
importance to technical aspects of a report. While the respondents ranked the degree of 
corroboration with previous findings and the technicality of a report high (2 and 3, respectively), 
they also ranked the representativeness of the sample as important and the validity of the 
evaluation design as least important, (1 1 and 13, respectively). It may be premature to drawany 
conclusions from such a small sample of political respondents. One could infer thatpoliticians 
either did not understand the survey, did not feel it was worthy attending to, or placed little 
importance on program evaluation reports as a factor in making politicaldecisions. If any of the 
above three speculations are true, program evaluators are in need of a professional lobbyist, in 
the opinion of the writer. 

A program evaluator can obtain the necessary training from institutions of higher learning. 
Although the requisite foundation courses and a majority of the technical courses are offered 
by university schools of education, some technical coursework may have to be taken in areas 
offered outside of schools of education. 

Because of the political aspects of educational program evaluation, an evaluator will probably 
learn to be sensitive to lay and professional individuals and groups and their anxieties about or 
hostility toward evaluators by his direct interactions with such personnel rather than by formal 
coursework. As one who has been labeled "educational middleman," (Bates, 1967) and one 
who has been cautioned not to take an absolutist position in his relationship to other 
educational experts, (Sorenson, 1968), an evaluator must learn the art of interacting as a 
philosopher-sage as well as a scientist-investigator. 
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Fortunately, it appears that the potential Tor upgrading the skills and experiential background 
for educational prograrrt evaluators is real. 

Knowing what may be deerred as desirable training, professional experiences, personal 
attributes, and performance criteria for educational program evaluators is not the same thing 
as knowing what practicing evaluators actually have had in the way of training and experience 
or what their professiona I behavior is like. Whether a study to determine the current state of the 
art for practicing evaluators would yield findings which would be reassuring or horrifying to 
member* of this association or client agencies is irrelevant. What is important is to devote as 
much time and energy to the careful training, education, and development of educational 
program evaluators as is devoted to the construction of complex, multi-dimensional matrices of 
educational program evaluation models. And after such pedagogical issues have boon tended 
to, it appears that activities to make the professional educational program evaluator and his 
product an important part of the political process need attending to, also. 
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